On Figures~\ref{fig:seq1} to~\ref{fig:seq3} we show an excerpt of an interaction
between the system and a DF. The figures show a 2D map from top view and the 3D
in-game view. In Figure~\ref{fig:seq1}, the user, represented by a blue
character, has just entered the upper left room. He has to push the button close
to the chair. The first candidate utterance selected is ``red closest to the
chair in front of you''. Notice that the referring expression uniquely identifies
the target object using the spatial proximity of the target to the chair. This
referring expression is generated without any reasoning on the target
distractors, just by considering the current state of the task plan and the user
position. 


%\begin{figure}
%\centering
%\begin{tabular}{cc}
%\subfloat[red closest to the chair in front of you]{\includegraphics[scale=0.22]{images/map1-screenshot.png}} &
%\subfloat[the closest one]{\includegraphics[scale=0.22]{images/map2-screenshot.png}} \\
%\subfloat[good]{\includegraphics[scale=0.22]{images/map2-1-screenshot.png}} &
%\subfloat[go back to the room with the lamp]{\includegraphics[scale=0.22]{images/map3-screenshot.png}}
%\end{tabular}
%\caption{Example interaction}
%\end{figure}

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.3]{images/map1-screenshot.png}
\end{center}
\caption{``red closest to the chair in front of you''}
\label{fig:seq1}
\end{figure}


\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.3]{images/map2-screenshot.png}
\end{center}
\caption{``the closet one''}
\label{fig:seq2}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.3]{images/map2-1-screenshot.png}
\end{center}
\caption{``good''}
\label{fig:seq2-1}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.3]{images/map3-screenshot-new.jpg}
\end{center}
\caption{``go back to the room with the lamp''}
\label{fig:seq3}
\end{figure}


After receiving the instruction the user gets closer to the button as shown in
Figure~\ref{fig:seq2}. As a result of the new user position, a new task plan
exists, the set of candidate utterances is recalculated and the system selects a
new utterance, namely ``the closet one''. 

The generation of the ellipsis of the button or the chair is a direct consequence
of the utterances normally said in the corpus at this stage of the task plan that
is, when the user is about to manipulate this object. From the point of view of
referring expression algorithms, the referring expression may not be optimal
because it is over-specified for instance a pronoun would be preferred as in
``click it''. Furthermore, the instruction contains a spelling error (`closet'
instead of `closest'). In spite of this non optimality, the instruction led our
user to execute the intended reaction, namely pushing the button.

Right after the user clicks on the button (Figure~\ref{fig:seq2-1}), the system
selects an utterance corresponding to the new task plan. The player position
stayed the same so the only change in the plan is that the button no longer needs
to be pushed. In this task state, DGs usually give acknowledgements and this is
then what our selection algorithm selects: ``good''.

After receiving the acknowledgement, the user turns around and walks forward, and
the next action in the plan is to leave the room (Figure~\ref{fig:seq3}).  The
system selects the utterance ``go back to the room with the lamp'' which refers
to the previous interaction.  Again, the system keeps no representation of the
past actions of the user, but such utterances are the ones that are found at this
stage of the task plan. 

We show in Figure~\ref{all-utts} all candidate utterances selected when exiting
the room in Figure~\ref{fig:seq3}. That is, for our system purposes, all the
utterances in the figure are paraphrases of the one that is actually uttered in
Figure~\ref{fig:seq3}.  As we explained in Section~\ref{selection}, the utterance
with the longest reaction is selected first (``go back to the room with the
lamp''), the second utterance with the longest reaction is selected second (``ok
go back again to the room with the lamp''), and so on. As you can observe in
Figure~\ref{all-utts} the utterances in the candidate set can range from
telegraphic style like ``L'' to complex sentences like ``Go through the opening
on the left with the yellow wall paper''. Several kinds of instructions are
displayed, acknowledgements such as ``yes'', pure moving instructions like
``left'' or ``straight'', instructions that refer to the local previous history
such as ``go back out the room'' or ``ok now go out the same door'' and
instructions that refer back to the global history such as ``okay now go back to
the original room''. 

Due to the lack of orientation consideration in our system, some orientation
dependent utterances are inappropriate in this particular context. For instance,
``\emph{left}'' is incorrect given that the player does not have to turn left but
go straight in order to go through the correct door.  However, most of the
instructions, even if quite different among themselves, could have been
successfully used in the context of Figure~\ref{fig:seq3}. 

\begin{figure*}
\begin{multicols}{2}
\begin{small}
\begin{tabular}{l l}
L &
go \\
yes &
left\\
straight &
now go back\\
go back out &
now go back out\\
closest the door &
down the passage\\
go back to the hallway &
nowin to the shade room\\
go back out of the room &
out the way you came in\\
exit the way you entered &
ok now go out the same door\\
back to the room with the lamp &
go back to the door you came in\\
Go through the opening on the left &
okay now go back to the original room\\
okay now go back to where you came from &
ok go back again to the room with the lamp\\
%now i ned u to go back to the original room & % too long
%Go through the opening on the left with the yellow wall paper\\ % too long
\end{tabular}
\end{small}
\end{multicols}
\caption{All candidate selected utterances when exiting the room in Figure~\ref{fig:seq3}}
\label{all-utts}
\end{figure*}




